Actively Estimating Crowd Annotation Consensus
نویسندگان
چکیده
منابع مشابه
Actively Estimating Crowd Annotation Consensus
The rapid growth of storage capacity and processing power has caused machine learning applications to increasingly rely on using immense amounts of labeled data. It has become more important than ever to have fast and inexpensive ways to annotate vast amounts of data. With the emergence of crowdsourcing services, the research direction has gravitated toward putting the wisdom of crowds to bette...
متن کاملOn Actively Teaching the Crowd to Classify
Is it possible to teach workers while crowdsourcing classification tasks? Amongst the challenges: (a) workers have different (unknown) skills, competence, and learning rate to which the teaching must be adapted, (b) feedback on the workers’ progress is limited, (c) we may not have informative features for our data (otherwise crowdsourcing may be unnecessary). We propose a natural Bayesian model...
متن کاملA framework to monitor, model, and actively manage crowd behaviour
In this poster we outline our framework for the analysis of crowds in different settings (city centres, sports events, etc.) and situations (regular or extreme). The framework has a monitor, a model and an intervene component to evaluate a number of strategies for risk and damage mitigation measures. This is illustrated with an example of a crowd at a football stadium.
متن کاملTemporal Information Annotation: Crowd vs. Experts
This paper describes two sets of crowdsourcing experiments on temporal information annotation conducted on two languages, i.e., English and Italian. The first experiment, launched on the CrowdFlower platform, was aimed at classifying temporal relations given target entities. The second one, relying on the CrowdTruth metric, consisted in two subtasks: one devoted to the recognition of events and...
متن کاملEffective Crowd Annotation for Relation Extraction
Can crowdsourced annotation of training data boost performance for relation extraction over methods based solely on distant supervision? While crowdsourcing has been shown effective for many NLP tasks, previous researchers found only minimal improvement when applying the method to relation extraction. This paper demonstrates that a much larger boost is possible, e.g., raising F1 from 0.40 to 0....
متن کاملذخیره در منابع من
با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید
ژورنال
عنوان ژورنال: Journal of Artificial Intelligence Research
سال: 2018
ISSN: 1076-9757
DOI: 10.1613/jair.5727